Skip to content

stats: Record per resource state file statistics in telemetry#5199

Open
shreyas-goenka wants to merge 1 commit into
mainfrom
shreyas-goenka/state-size-telemetry
Open

stats: Record per resource state file statistics in telemetry#5199
shreyas-goenka wants to merge 1 commit into
mainfrom
shreyas-goenka/state-size-telemetry

Conversation

@shreyas-goenka
Copy link
Copy Markdown
Contributor

@shreyas-goenka shreyas-goenka commented May 6, 2026

This PR adds tracking for state file sizes per resources we'll see in DMS. This is important to know - to design and prioritize work to integrate with UC internal storate as a overflow mechanism for DMS state.

@shreyas-goenka shreyas-goenka force-pushed the shreyas-goenka/state-size-telemetry branch from 4a3106b to 715f019 Compare May 6, 2026 18:07
@shreyas-goenka shreyas-goenka force-pushed the shreyas-goenka/state-size-telemetry branch from 715f019 to 77ec7bc Compare May 6, 2026 18:32
@shreyas-goenka shreyas-goenka force-pushed the shreyas-goenka/state-size-telemetry branch from 77ec7bc to 85dd380 Compare May 6, 2026 18:49
@shreyas-goenka shreyas-goenka force-pushed the shreyas-goenka/state-size-telemetry branch from 85dd380 to 65bb595 Compare June 2, 2026 09:41
Adds a typed `resources_metadata` field to the bundle deploy telemetry
event, capturing per resource type:
- count of resources declared in the bundle configuration (replacing the
  deprecated DatabricksBundleDeployEvent.resource_*_count fields, still
  populated during the transition)
- max / mean / median state size in bytes across resources of that type

plus the whole simulated state file size and the effective deploy engine
("direct" or "terraform").

State sizes are computed from the bundle configuration, not by reading the
on-disk state file. Each resource's typed config is run through the direct
engine's adapter.PrepareState -- the same transformation direct uses to
derive the value it persists to resources.json -- and marshaled with the
indented encoding dstate.SaveState uses. Two consequences:
- the numbers are engine-independent, so direct and terraform deploys of the
  same logical bundle report identical sizes (tfstate is never read);
- for direct deploys each per-resource size equals len(entry.State) on disk
  byte-for-byte, and the file size matches resources.json up to the
  Lineage/Serial fields.

The feature is one isolated module (bundle/phases/resources_metadata.go) plus
one line at the telemetry-emission call site. Telemetry never fails a deploy:
all parse/adapter errors are logged at debug level and treated as missing
data.

The universe proto (DatabricksBundleDeployEvent.resources_metadata,
BundleResourcesMetadata, ResourceMetadata) already exists, so this is ingested
rather than dropped.

Co-authored-by: Isaac
@shreyas-goenka shreyas-goenka force-pushed the shreyas-goenka/state-size-telemetry branch from 65bb595 to 44e4bc0 Compare June 2, 2026 10:10
@shreyas-goenka shreyas-goenka changed the title Track bundle resource counts and state file sizes in telemetry stats: Record per resource state file statistics in telemetry Jun 2, 2026
@shreyas-goenka shreyas-goenka marked this pull request as ready for review June 2, 2026 10:14
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 2, 2026

Approval status: pending

/acceptance/bundle/ - needs approval

Files: acceptance/bundle/telemetry/deploy/out.telemetry.txt, acceptance/bundle/telemetry/test.toml
Suggested: @denik
Also eligible: @andrewnester, @pietern, @janniklasrose, @lennartkats-db, @anton-107

/bundle/ - needs approval

Files: bundle/phases/resources_metadata.go, bundle/phases/resources_metadata_test.go, bundle/phases/telemetry.go
Suggested: @denik
Also eligible: @andrewnester, @pietern, @janniklasrose, @lennartkats-db, @anton-107

/libs/telemetry/ - needs approval

Files: libs/telemetry/protos/bundle_deploy.go
Suggested: @simonfaltum
Also eligible: @renaudhartert-db, @hectorcast-db, @parthban-db, @tanmay-db, @Divyansh-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @simonfaltum, @renaudhartert-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot
Copy link
Copy Markdown
Collaborator

Commit: 44e4bc0

Run: 26813106573

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants